The realist evaluation of educational technology

PURPOSE. This thesis considers the best way to address the challenges faced by educators, institutions and funding bodies trying to not only develop and implement educational technology successfully but tackle the challenge of understanding and evidencing what works (and what does not) and why. The aim of the research was to find and validate an evaluation method that provided usable and useful evidence. APPROACH. A range of evaluations were undertaken to elicit the strengths and weaknesses of different approaches, augmented by drawing upon the experiences and outcomes published by others. An analysis of the issues was made and significance of the problem established. The problem being premature timing, unsuitable models, rapid change, complex implementation chains, inconsistent terminology, ideology and marketisation. A tailored realist evaluation framework was proposed as an alternative method and it was tested to evaluate an institutional lecture capture (LC) initiative. FINDINGS. The theory-driven realist approach provided a level of abstraction that helped gather evidence about wider influences and theories of potential future impact of the LC programme and its linked policy. It proved valuable in generating real and practical recommendations for the institution, including what more could be done to improve uptake and support embedding in teaching and learning, from practice, policy and technological points of view. It identified some unanticipated disadvantages of LC as well determining how and when it was most effective. PRACTICAL IMPLICATIONS. A Realist Evaluation of Technology Initiative (RETI) framework has been produced as tool to aid the rapid adoption of the approach. Recommendations for future research and seven guiding principles have been proposed to encourage the formation of a community of realist evaluative researchers in educational technology. ORIGINALITY/VALUE. The rigorous application of a tailored realist evaluation framework (RETI) for educational technology (including the development of two Domain Reference Models) is the primary contribution to new knowledge. This research is significance because it has potential to enable the synthesis of evaluation findings within the sector. This will enable an evidence-base of what works, for whom, in which contexts and why, ultimately benefiting policy-makers and practitioners to support better informed decision making and investment in education

Co-Tutor: a relationship management system to enable staff to monitor students' engagement and provide support to at risk students

Co-Tutor: a relationship management system to enable staff to monitor students' engagement and provide support to at risk student

Supporting and developing engineering education in the UK and beyond

This paper details the emergence and development of the 'Centre for Engineering and Design Education' (CEDE) at Loughborough University, UK, and provides ample evidence that such a Centre can prove to be a highly effective support mechanism for discipline-specific academics and can develop and maintain valuable national and international networks and collaborations along with considerable esteem for the host university. The CEDE is unique in the UK and has achieved considerable success and recognition within the local engineering education community and beyond. Here we discuss the historical background of the Centre's development, the context in which it operates, and its effective management and operation strategy. The success it has enjoyed is described through examples, with much evidence of the generation of a significant amount of external funding; the development of high quality learning spaces; learning technology systems, improvements in curriculum design; a strong record of research and publication on the pedagogy of engineering; strong links with industry and employers; and a wealth of connections and know-how built up over the years

Evaluating business simulation software: approach, tools and pedagogy

Purpose - To identify a business simulation appropriate for MEng Engineering students. The selection was based on the following factors; exploring methods for evaluating potential software and enhancing the learner experience. Design/methodology/approach - An interdisciplinary project team was formed to try and resolve the pedagogic, technical and business aspects that would need to be addressed in order to implement such software within the programme. Tools included a questionnaire to assess the potential enhancement of employability skills and a usability questionnaire on ease of use. These were supplemented with discourse on technical and pedagogic issues. Findings - After the initial scoping study, our findings indicated that two business simulation software packages had potential. These were ‘Marketplace - Venture Strategy’ and ‘SimVenture’. Marketplace proved to be the most suitable in terms of the pedagogic and technical requirements. Research limitations / implications - We were not able to fully trial each simulation over the recommended duration of play because of practical time constraints and we did not have any student contribution to the process. Findings will need to be verified with the piloting cohort of students. Further pedagogic research could be carried out to evidence the enhancement to the student learning experience. Originality/value - This study is valuable because it purposefully uses an interdisciplinary team comprising expertise in; teaching and learning, technology, business and sector knowledge. This was vital in the decision making process. Development of generic methods and tools to measure and evaluate software suitability in relation to usability and employability skills

The grass-roots development and institutional embedding of the tutoring management system Co-Tutor

The grass-roots development and institutional embedding of the tutoring management system Co-Tuto

Exploring the balance between automation and human intervention in improving final year university student non-completion

This paper examines the research methods used in the 'Pedestal for Progression' project, a project that set out to determine why students at Loughborough University fail to complete their final year. It demonstrates how methods adopted can be used to enhance student experience and improve retention. Initial research with students found that the difference between student experience in initial and final years can be characterised by concern over the independent study required for the dissertation project and associated worries of managing workloads with competing deadlines. Interviews and workshops with students also identified a wider concern about the quality of relationships with technical, administrative and academic staff. In addition, research found that the final year can be flooded by concerns over employability. Fundamental to these issues are student relationships. Using the methods of Service Design and Data Mining the project designed, implemented and assessed a number of initiatives aimed at alleviating these student concerns. Key to the theory of Service Design is the management of points of contact with service providers and vital to Data Mining is the identification of patterns of behaviour that could predict non-completion. Service Design aims to provide customer focused highly desirable services. Whilst, on the other hand, data mining aims to identify signals that determine those at risk of not completing courses. This paper examines the use of Service Design and Data Mining in Higher Education from the results of the project and determined that whilst the methods can be used in a complementary manner, each derives from different paradigms of knowledge

A platform for digital innovation in the management and administration of assessment and feedback

This paper outlines a series of key developments and interventions that have enabled a platform for digital innovation in the management and administration of the assessment and feedback processes within a top 10 UK University. The joined up platform of technological solutions and associated transformative staff practices demonstrated a wide impact on efficiency and productivity across the organisation. The initiative adopted new approaches to innovation and change to help deal with two major challenges: 1. Addressing the technical, organisational and cultural complexity of such a programme of work. 2. Addressing the challenge of limited resources with which to deliver lean, practical and useful digital solutions that can be efficiently and effectively adopted by staff and scaled up across the institution. Our challenge: Members of staff use a disparate set of learning technologies and desktop software to create and deliver assessments and to mark work and provide feedback to students. Each department and School tended to operate using a different set of procedures and forms to issue assessments and collect submissions, register participation and provide feedback. There was no clear and consistent mechanism to return work back to students or to check the consistency and quality of feedback. Individual practices and attitudes varied and academic staff needed to be convinced that new ways of working would be better for themselves and their students, especially with regards to adopting and accepting new technologies. Our approach: This paper describes the pragmatic and agile way in which a small team set out to try and improve things one step at a time with the limited resources that were available to them. This initiative was a community driven activity taking an agile and very lean approach to change. It was recognised that changes to pedagogic and administrative practices, as well as development of a variety of digital solutions would be required. The solution: The digital platform developed (containing 5 components) comprises; features for assessment planning; communication; collecting data on submissions; collating data on feedback and returns; using the data to provide actionable insights in student engagement and staff practices; support a feedback dialogue between multiple staff and students; create a digital platform architecture underpinning multiple assessments, submissions and feedback types and applications; the creation of an assessment and submission data standard in order to join up data in disparate systems. Digital innovation: All staff embraced new social and technical methodologies to address pedagogic practices and technical hurdles. 2014/15 marked a significant year in the process of development and adoption. The CASPA application became a mature, useful and highly used product. The roll-out of stand-alone kiosks and integration of assessment and submission data into the tutoring system Co-Tutor, was so successful, that adoption went from 4 departments to 9 in one year with minimal need for advocacy or support. Systems integration work via a data warehouse, creating a taxonomy for assessments, submissions and feedback data, is innovative in the sector allowing multiple learning technologies to ‘talk to each other’ effectively. This paper will be of interest to practitioners and IT staff who are involved in the development, implementation and embedding of systems and process in their own universities

A blueprint for success: a model for developing engineering education in the UK

This paper details the emergence and development of the ‘Centre for Engineering and Design Education’ (CEDE) at Loughborough University, UK, and provides a blueprint for success. With ample evidence that such a Centre can prove to be a highly effective support mechanism for discipline-specific academics and can develop and maintain valuable national and international networks and collaborations along with considerable esteem for the host university. The CEDE is unique in the UK and has achieved considerable success and recognition within the local engineering education community and beyond for the past 16 years. Here we discuss the historical background of the Centre’s development, the context in which it operates, and its effective management and operation strategy. The success it has enjoyed is described through examples, with much evidence of the generation of a significant amount of external funding; the development of high quality learning spaces; learning technology systems, open source software and improvements in curriculum design; a strong record of research and publication on the pedagogy of engineering; strong links with industry and employers; and a wealth of connections and know-how built up over the years. This paper provides the institutions with a model blueprint for success in developing engineering education

An evaluation of business simulation games for the Management module of the MEng Aeronautical Engineering degree at Loughborough University

There is a drive within engineering disciplines at Loughborough University to develop the employability skills of undergraduate students. The engCETL (Engineering Centre for Excellence in Teaching & Learning) has a broad remit to enhance links with industry and to underpin developments in learning and teaching with pedagogic research and technology development. The Centre does this through research and development projects that are proposed by academics within the engineering related departments and carried out in conjunction with specialists from the engCETL team. Prof Rob Thring, Head of the Aeronautical and Automotive Engineering Department proposed a project to the engCETL. His requirement was for some form of business simulation software for the undergraduates to use as part of the Management module on the MEng programme. Currently the students come up with an idea for a new business, create a business plan for the venture and take part in a ‘Dragons’ Den’ style presentation at the end of the project to representatives from the department and industry. However, the department would like to take this project a step further and provide the students with the opportunity to take part in a simulated business environment where they could explore the idea of setting up or running a business as close to the real world as possible. The intention would be to enliven and enrich the student’s learning experience with skills development such as; enterprise, leadership, management, teamwork, fiscal sense, business judgement and inventiveness amongst others. An interdisciplinary project team was formed to try and resolve the pedagogic, technical and business aspects that would need to be addressed in order to implement such software within the MEng programme. The approach taken has been to form a set of criteria based on certain curriculum requirements but keep the brief broad and carry out a scoping study of existing software (commercial and open source) and take account of the academic literature in this area. After the initial scoping study, our findings indicate two commercial business simulations that have potential for use on the course. These were; ‘Marketplace Simulation’ (http://www.marketplace-simulation.co.uk) and SimVenture (http://www.simventure.co.uk). An in-depth evaluation was then carried out for the two simulations. This evaluation comprised two teams made up of academics, industrial representatives and engCETL staff. The software was thoroughly examined in terms of what each application could offer to the learning experience of the students, resources to support staff and the costs involved, for example, staff time in embedding the software into the curriculum.This paper will highlight the approach taken, findings and recommendations from the evaluation of the two business simulations. The recommendations will be presented in the context of all engineering disciplines and will cover; appropriateness of the chosen software for the programme level, plans for embedding into the curriculum, potential learning outcomes and assessment methods. It will benefit all those interested in methods for evaluating potential simulation games for suitability within the curriculum and the development of enterprise and employability skills

A PLATFORM FOR DIGITAL INNOVATION IN THE MANAGEMENT AND ADMINISTRATION OF ASSESSMENT AND FEEDBACK

This paper outlines a series of key developments and interventions that have enabled a platform for digital innovation in the management and administration of the assessment and feedback processes within a top 10 UK University. The joined up platform of technological solutions and associated transformative staff practices demonstrated a wide impact on efficiency and productivity across the organisation. The initiative adopted new approaches to innovation and change to help deal with two major challenges: 1. Addressing the technical, organisational and cultural complexity of such a programme of work. 2. Addressing the challenge of limited resources with which to deliver lean, practical and useful digital solutions that can be efficiently and effectively adopted by staff and scaled up across the institution. Our challenge: Members of staff use a disparate set of learning technologies and desktop software to create and deliver assessments and to mark work and provide feedback to students. Each department and School tended to operate using a different set of procedures and forms to issue assessments and collect submissions, register participation and provide feedback. There was no clear and consistent mechanism to return work back to students or to check the consistency and quality of feedback. Individual practices and attitudes varied and academic staff needed to be convinced that new ways of working would be better for themselves and their students, especially with regards to adopting and accepting new technologies. Our approach: This paper describes the pragmatic and agile way in which a small team set out to try and improve things one step at a time with the limited resources that were available to them. This initiative was a community driven activity taking an agile and very lean approach to change. It was recognised that changes to pedagogic and administrative practices, as well as development of a variety of digital solutions would be required. The solution: The digital platform developed (containing 5 components) comprises; features for assessment planning; communication; collecting data on submissions; collating data on feedback and returns; using the data to provide actionable insights in student engagement and staff practices; support a feedback dialogue between multiple staff and students; create a digital platform architecture underpinning multiple assessments, submissions and feedback types and applications; the creation of an assessment and submission data standard in order to join up data in disparate systems. Digital innovation: All staff embraced new social and technical methodologies to address pedagogic practices and technical hurdles. 2014/15 marked a significant year in the process of development and adoption. The CASPA application became a mature, useful and highly used product. The roll-out of stand-alone kiosks and integration of assessment and submission data into the tutoring system Co-Tutor, was so successful, that adoption went from 4 departments to 9 in one year with minimal need for advocacy or support. Systems integration work via a data warehouse, creating a taxonomy for assessments, submissions and feedback data, is innovative in the sector allowing multiple learning technologies to ‘talk to each other’ effectively. This paper will be of interest to practitioners and IT staff who are involved in the development, implementation and embedding of systems and process in their own universities